Add playback tests as preflight step 8 and fix test robustness#7506
Add playback tests as preflight step 8 and fix test robustness#7506jongio merged 2 commits intoAzure:mainfrom
Conversation
Add new preflight check that discovers and runs functional tests with existing recordings in AZURE_RECORD_MODE=playback. This catches stale test recordings before they cause CI failures. Changes: - Add PlaybackTests() standalone mage target and preflight step 8 - Add discoverPlaybackTests() to scan testdata/recordings for test names - Add setEnvScoped() helper for safe env var save/restore with LookupEnv - Add goModVersion() to pin GOTOOLCHAIN and prevent parallel build races - Add excludedPlaybackTests map for known stale recordings - Use regexp.QuoteMeta for test name regex safety - Filter non-test directories with strings.HasPrefix guard - Modernize os.IsNotExist to errors.Is(err, fs.ErrNotExist) - Delegate runStreaming to runStreamingWithEnv to eliminate duplication - Fix kubectl test to use mock CommandRunner (no real kubectl needed) - Fix rzip tests with conditional symlink handling (no skip) - Increase grpcbroker cancellation timeout from 1s to 5s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new “playback” functional test run to mage preflight to catch stale/invalid recordings locally (before CI), and makes several unit tests more robust across developer environments and under load.
Changes:
- Add preflight step 8 + standalone mage target to discover recording-backed functional tests and run them in
AZURE_RECORD_MODE=playback. - Refactor env var save/restore into a shared helper and add optional
GOTOOLCHAINpinning fromgo.mod. - Improve test robustness: mock kubectl in kube config tests, make rzip symlink tests OS-capability-aware, and relax a cancellation timing assertion.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| cli/azd/magefile.go | Adds playback-test discovery/execution to preflight + new env helpers/toolchain pinning. |
| cli/azd/pkg/tools/kubectl/kube_config_test.go | Avoids requiring a real kubectl binary by mocking the command runner. |
| cli/azd/pkg/rzip/rzip_test.go | Makes symlink-related expectations conditional on platform symlink support. |
| cli/azd/pkg/grpcbroker/message_broker_test.go | Increases cancellation test timeout to reduce flakiness under load. |
- Fix -run regex to match subtests by changing anchor from $ to (/|$) - Update Preflight doc comment to mention step 8 (playback tests) - Fix symlinkAvailable comment to not reference a dir parameter - Only set GOTOOLCHAIN when not already set to respect user overrides
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
…rsion management - Update Go prerequisite from 1.26 to 1.26.1 (Azure#7488) - Document new `mage playbackTests` target added as preflight step 8 (Azure#7506) - Add Go Version Management section documenting Update-GoVersion.ps1 script and validate-go-version CI workflow (Azure#7488) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Adds a new preflight check (step 8) that discovers and runs functional tests with existing recordings in
AZURE_RECORD_MODE=playback. This catches stale test recordings before they cause CI failures, addressing the issue described in #7251 where CI breaks due to outdated recordings that developers don't notice locally.Motivation
From this comment: CI failures are often caused by stale test recordings, not flaky tests. Developers currently have no local signal that their changes broke a recording until CI fails. This preflight step provides that signal.
What's Changed
Preflight Step 8: Playback Tests (
cli/azd/magefile.go)PlaybackTests()— Standalone mage target and new preflight step 8discoverPlaybackTests()— Scanstest/functional/testdata/recordings/for.yamlfiles/directories, extracts top-level test namessetEnvScoped()— Helper for safe env var save/restore usingos.LookupEnv(replaces 5 duplicated save/restore blocks)goModVersion()— Reads Go version fromgo.modto pinGOTOOLCHAINand prevent parallel build racesexcludedPlaybackTests— Map of known stale recordings to exclude (currentlyTest_CLI_Deploy_SlotDeployment)regexp.QuoteMetafor test name regex safetystrings.HasPrefix(name, 'Test')guardrunStreamingtorunStreamingWithEnvto eliminate duplicationTest Robustness Fixes
kube_config_test.go—Test_MergeKubeConfignow usesmockContext.CommandRunnerinstead of requiring realkubectlbinary, matching the mock pattern used by all other tests in the packagerzip_test.go— Symlink tests usesymlinkAvailable()probe and conditionally adjust expectations instead of skipping. Tests always pass regardless of Windows developer mode statusmessage_broker_test.go— Increased context cancellation test timeout from 1s to 5s for robustness under loadHow It Works
testdata/recordings/for.yamlcassette files.yamlsuffix, takes prefix before first.for variant recordings)AZURE_RECORD_MODE=playback— zero Azure credentials needed, all HTTP responses replayed from diskTesting
mage preflight— All 8 steps pass, zero skips, zero warnings-shortflagkubectl, no symlink support required)